Weighing Weights

Have you ever…

Have you ever…

Model Selection

The Problem

• You have:
  • one realization
  • different data sources
  • coded multiple competing hypothesis
• You want to know
  • Which one is right?

Naive Error Minimisation

• Compute summary statistics
  • Real data \(\theta\)
  • Model \(\hat \theta\)
• Pick model which minimizes difference in summary statistics: \[ \min \sum | \theta-\hat \theta| \]
• Weighing:
  • Correlation
  • Variance
  • Units

Data Available - Visit Counts

Summary Statistics - 1

Data Available - Logbook

Summary Statistics - 2

• Fit Random Utility Model \[ \text{Pr}(\text{Choice}=1) = \frac{e^{\beta_i x_i}}{\sum e^{\beta_j x_j}} \]
• Fit to data by logit
• The \(\beta\) are your summary statistics
• Classic indirect inference

Data Available - Aggregate Information

Summary Statistics - Model and Reality

Fishery Example

• 9 hypotheses:
  • each represents a different behaviour algorithm
• 3 scenarios:
  • geographical/technical differences

Fishery Naive Error

Fishery Naive Error 2

The Solution

1. Generate summary statistics \(\theta_1,\dots,\theta_n\)
2. Compute distance from data as weighted distance between summary statistics \[ \sum w_i \Delta_{\theta_i} \] \[ f(\Delta_{\theta_1},\dots,\Delta_{\theta_n}) \]
3. Tune \(w_i\) to maximize model selection success in training data-sets

Build Training Data

• If we generate data with one model, can we select it back by minimizing naive error?
• For each hypothesis:
  • generate 100 test-cases
  • for each test case:
    • generate one run for each hypothesis
    • compute summary statistics distances
    • pick model which minimizes it

Training Data

Training Data - 2

Training Data - 3

• Turn this into a classifier problem
• Find function predicting: \[ f(\Delta_{\theta_1},\dots,\Delta_{\theta_n}) \to \text{Pr}(\text{Correct}) \]
• Pick hypothesis by \[ \max f(\Delta_{\theta_1},\dots,\Delta_{\theta_n})\]

Results

Parameter Errors

• Parameter uncertainties
• If we get some parameters wrong, can we still pick the correct model?
• Biology mis-specification
• Add mistakes to your training set

Parameter Errors - Results

Calibration

• Sometimes you don’t have discrete hypotheses
• Continuous parameters \(x\) you want to minimise \[ \arg_x \min \sum w_i \Delta_{\theta_i}(x) \]

1D Calibration Example

The Problem

\[ \arg_x \min \sum w_i \Delta_{\theta_i}(x) \] * Can we change \(w\) to make minimization easier/better?

What we would like?

The Solution

• Generate many examples in pairs, varying parameters \(x\)
• Solve as a regression problem

Training Data - 1

Training Data - 2

Regression Problem

• Try to predict \(\Delta_x\) by looking at difference in summary statistics \[ \Delta_x \sim g(\Delta_{\theta_1},\dots,\Delta_{\theta_n}) \]

The Result

In practice

In pratice - 2